<!DOCTYPE html>
<html lang="en">

<head><meta name="generator" content="Hexo 3.9.0">
	<!-- hexo-inject:begin --><!-- hexo-inject:end --><meta http-equiv="content-type" content="text/html; charset=utf-8">
	<meta content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=0" name="viewport">
	
	<!-- title -->
	
	<title>
	
		linux工具之awk | 
	 
	稻草人的编程之路
	</title>
	
	<!-- keywords,description -->
	 
		<meta name="description" content="人生的意义，不在于最终获得了什么，而在于曾经努力追求过什么">
	

	<!-- favicon -->
	
	<link rel="shortcut icon" href="/favicon.ico">
	
  

	<link rel="stylesheet" href="/css/main.css">
	<link rel="stylesheet" href="https://cdn.staticfile.org/font-awesome/4.7.0/css/font-awesome.min.css">
	<link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/highlightjs/cdn-release@9.17.1/build/styles/darcula.min.css">

	<script src="https://cdn.jsdelivr.net/gh/highlightjs/cdn-release@9.17.1/build/highlight.min.js"></script>
	<script src="https://cdn.jsdelivr.net/npm/jquery@3.4.1/dist/jquery.min.js"></script>
	<script src="https://cdn.jsdelivr.net/npm/jquery-pjax@2.0.1/jquery.pjax.min.js"></script>
	<script src="/js/main.js"></script>
	
		<script src="https://cdn.jsdelivr.net/npm/leancloud-storage/dist/av-min.js"></script>
		<script src="https://cdn.jsdelivr.net/npm/valine@1.3.10/dist/Valine.min.js"></script>
	
	
		<script async src="//busuanzi.ibruce.info/busuanzi/2.3/busuanzi.pure.mini.js"></script><!-- hexo-inject:begin --><!-- hexo-inject:end -->
	
</head>

<body>
	<!-- hexo-inject:begin --><!-- hexo-inject:end --><header id="header">
    <a id="title" href="/" class="logo">稻草人的编程之路</a>

	<ul id="menu">
		<li class="menu-item">
			<a href="/about" class="menu-item-link">ABOUT</a>
		</li>
		
		<li class="menu-item">
			<a href="https://github.com/wangxiaochuang" class="menu-item-link" target="_blank">
				<i class="fa fa-github fa-2x"></i>
			</a>
		</li>
	</ul>
</header>

	
<div id="sidebar">
	<button id="sidebar-toggle" class="toggle" ><i class="fa fa-arrow-right " aria-hidden="true"></i></button>
	
	<div id="site-toc">
		<input id="search-input" class="search-input" type="text" placeholder="search...">
		<div id="tree">
			

			
							<ul>
								<li class="directory">
									<a href="#" class="directory">
										<i class="fa fa-plus-square-o"></i>
										linux
									</a>
									
							<ul>
								<li class="directory">
									<a href="#" class="directory">
										<i class="fa fa-plus-square-o"></i>
										tools
									</a>
									
							<ul>
								<li class="file active">
									<a href="/2019/08/18/100.html">
										awk
									</a>
								</li>
								<div class="article-toc" style="display: none;"></div>
							</ul>
			
							<ul>
								<li class="file">
									<a href="/2019/08/18/102.html">
										grep
									</a>
								</li>
								<div class="article-toc" style="display: none;"></div>
							</ul>
			
							<ul>
								<li class="file">
									<a href="/2019/08/18/101.html">
										sed
									</a>
								</li>
								<div class="article-toc" style="display: none;"></div>
							</ul>
			
								</li>
								
							</ul>
			
								</li>
								
							</ul>
			
		</div>
	</div>
</div>

	<!-- 引入正文 -->
	<div id="content">
		<h1 id="article-title">

	awk
</h1>
<div class="article-meta">
	
	<span>wangxiaochuang</span>
	<span>2019-08-18 08:04:33</span>
    
		<div id="article-categories">
            
                
                    <span>
                        <i class="fa fa-folder" aria-hidden="true"></i>
                        <a href="/categories/linux/">linux</a>
						
							>
						
                    </span>
                
            
                
                    <span>
                        <i class="fa fa-folder" aria-hidden="true"></i>
                        <a href="/categories/linux/tools/">tools</a>
						
                    </span>
                
            
		</div>
    
</div>

<div id="article-content">
	<h1 id="awk"><a class="markdownIt-Anchor" href="#awk"></a> awk</h1>
<p>awk不仅仅是linux系统上的一个命令，它更是一种编程语言，可以用来处理数据和生成报告</p>
<p>awk处理的数据来源可以是文件，也可以是标准输入；其可以在命令行上直接编辑命令进行操作，同时还可以写成程序去处理复杂的任务</p>
<h2 id="基本用法"><a class="markdownIt-Anchor" href="#基本用法"></a> 基本用法</h2>
<p>有数据文件内容如下：</p>
<blockquote>
<p>jack, male, 28, beijing<br>
yuanyuan, female, 26, chongqing<br>
xiao, female, 26, sichuan<br>
straw, male, 25, shanghai<br>
lili, female, 23, chongqing</p>
</blockquote>
<p>例子：</p>
<pre class="highlight"><code class>CMD:
  cat data.txt |awk -F&quot;[, ]+&quot; '$2 ~ /female/ {print NR,$3,$NF}'
OUTPUT:
  2 26 chongqing
  3 26 sichuan
  5 23 chongqing
</code></pre>
<p>awk命令构成</p>
<pre class="highlight"><code class>awk [options] 'pattern {action}' file
</code></pre>
<p>其中<code>-F&quot;[, ]+&quot;</code>是选项，<code>$2 ~ /female/</code>是模式，<code>{print NR,$3,$NF}</code>是动作；</p>
<h3 id="域和记录"><a class="markdownIt-Anchor" href="#域和记录"></a> 域和记录</h3>
<h4 id="域"><a class="markdownIt-Anchor" href="#域"></a> 域</h4>
<p>awk中默认将一个记录按空格作为分隔符来分割为多个域，可以使用选项<code>-F</code>指定自定义域分隔符</p>
<p>分割为多个域后通过<code>$</code>来访问，如果将一个记录分割后的域当做数组的话，那这个数组是从1开始计数的；<code>$1</code>表示第一个域，<code>$2</code>表示第二个域，…，<code>$NF</code>表示最后一个域；如果要访问整行记录使用<code>$0</code></p>
<p>相关变量</p>
<table>
<thead>
<tr>
<th>变量</th>
<th>含义</th>
<th>说明</th>
</tr>
</thead>
<tbody>
<tr>
<td>NF</td>
<td>number of field，域总数</td>
<td>可以通过<code>$NR</code>访问最后一个域</td>
</tr>
<tr>
<td>FS</td>
<td>输入域分隔符</td>
<td>可以通过<code>BEGIN{FS=&quot;[, ]+&quot;}</code>达到与<code>-F</code>参数同样的效果，如：<code>cat data.txt |awk 'BEGIN{FS=&quot;[, ]+&quot;} $2 ~ /^f/ {print $1,$3}'</code></td>
</tr>
<tr>
<td>OFS</td>
<td>输出域分隔符</td>
<td>与FS类似，也是在BEGIN阶段设置</td>
</tr>
</tbody>
</table>
<h4 id="记录"><a class="markdownIt-Anchor" href="#记录"></a> 记录</h4>
<p>awk中默认将每一行作为一个记录，意思是awk会按换行符将文件的内容分为多个记录，拿上面的例子，每个人的所有信息为一个记录。在模式与动作中可以使用变量<code>NR</code>，这个变量表示当前处理的记录号，默认记录分隔符的话就应该是行号</p>
<p>相关变量</p>
<table>
<thead>
<tr>
<th>变量</th>
<th>含义</th>
<th>说明</th>
</tr>
</thead>
<tbody>
<tr>
<td>NR</td>
<td>number of record，当前记录的计数</td>
<td>默认是行号</td>
</tr>
<tr>
<td>RS</td>
<td>记录的分隔符</td>
<td>默认是换行符</td>
</tr>
<tr>
<td>ORS</td>
<td>输出域分隔符</td>
<td>默认是换行符</td>
</tr>
</tbody>
</table>
<h3 id="action"><a class="markdownIt-Anchor" href="#action"></a> action</h3>
<p>上面例子中的<code>print</code>是动作，表示要打印到标准输出，这是比较常见的动作，经常会使用。</p>
<p>这里的动作将应用于当前的记录</p>
<h3 id="pattern"><a class="markdownIt-Anchor" href="#pattern"></a> pattern</h3>
<p>这里使用<code>~</code>来进行正则匹配，表示第2列满足后面的正则表达式的记录才执行后面的动作，支持的匹配方法：</p>
<table>
<thead>
<tr>
<th>符号</th>
<th>示例</th>
<th>解释</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>~</code></td>
<td><code>$2 ~ /female/</code></td>
<td>匹配<code>//</code>中的正则表达式的记录才会执行动作</td>
</tr>
<tr>
<td><code>!~</code></td>
<td><code>$2 ~ /male/</code></td>
<td>不匹配<code>//</code>中的正则表达式的记录才会执行动作</td>
</tr>
<tr>
<td><code>&gt;</code></td>
<td><code>$3 &gt; 25</code></td>
<td>年龄大于25的动作才执行动作</td>
</tr>
<tr>
<td><code>&lt;</code></td>
<td><code>$3 &lt; 26</code></td>
<td>年龄小于26的动作才执行动作</td>
</tr>
<tr>
<td><code>==</code></td>
<td><code>$3 == 28</code></td>
<td>年龄为28的记录才执行动作</td>
</tr>
<tr>
<td><code>&amp;&amp;</code></td>
<td><code>$3 &gt; 23 &amp;&amp; $3 &lt; 28</code></td>
<td>多条件同时成立</td>
</tr>
<tr>
<td><code>||</code></td>
<td><code>$3 &lt; 25 || $3 &gt; 26</code></td>
<td>多条件求并集</td>
</tr>
</tbody>
</table>
<p>关于正则表达式常见符号</p>
<table>
<thead>
<tr>
<th>符号</th>
<th>解释</th>
<th>示例</th>
<th>说明</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>^</code></td>
<td>匹配开头</td>
<td>如：<code>$2 ~ /^f/</code></td>
<td></td>
</tr>
<tr>
<td><code>$</code></td>
<td>匹配结尾</td>
<td>如：<code>$1 ~ /w$/</code></td>
<td></td>
</tr>
<tr>
<td><code>{m,n}</code></td>
<td>匹配数量</td>
<td><code>$1 ~ /^[a-z]{4,5}$/</code></td>
<td>某些awk版本可能需要参数<code>--posix</code></td>
</tr>
</tbody>
</table>
<h3 id="options"><a class="markdownIt-Anchor" href="#options"></a> options</h3>
<table>
<thead>
<tr>
<th>参数</th>
<th>解释</th>
<th>示例</th>
<th>说明</th>
</tr>
</thead>
<tbody>
<tr>
<td>-F</td>
<td>指定域分隔符</td>
<td><code>-F&quot;[, ]+&quot;、-F:</code></td>
<td>默认为空格</td>
</tr>
<tr>
<td>–posix</td>
<td>使用posix标准正则表达式</td>
<td></td>
<td></td>
</tr>
</tbody>
</table>
<h3 id="awk工作方式"><a class="markdownIt-Anchor" href="#awk工作方式"></a> awk工作方式</h3>
<ol>
<li>按行分割符读取一个记录到缓存区</li>
<li>按域分隔符将该记录分割为一个一个域</li>
<li>如果存在pattern部分，进入第4步；如果不存在pattern部分，直接进入第5步</li>
<li>如果满足pattern的条件，进入第5步；如果不满足pattern的条件回到第1步</li>
<li>执行动作</li>
</ol>
<h2 id="进阶用法"><a class="markdownIt-Anchor" href="#进阶用法"></a> 进阶用法</h2>
<h3 id="begin与end模块"><a class="markdownIt-Anchor" href="#begin与end模块"></a> BEGIN与END模块</h3>
<h4 id="begin模块"><a class="markdownIt-Anchor" href="#begin模块"></a> BEGIN模块</h4>
<p>awk读取文件前就执行BEGIN模块的代码，一般用来定义内置变量的值，比如设置记录分隔符RS、ORS、域分隔符FS、ORS、自定义变量等</p>
<pre class="highlight"><code class>CMD:
  cat data.txt |awk 'BEGIN{FS=&quot;[, ]+&quot;;OFS=&quot;:&quot;} {print $1,$3}'
OUTPUT:
  jack:28
  yuanyuan:26
  xiao:26
  straw:25
  lili:23

# 如果有BEGIN阶段还有pattern条件
CMD:
  cat data.txt |awk 'BEGIN{FS=&quot;[, ]+&quot;;OFS=&quot;:&quot;} $3 &gt; 26 {print $1,$3}'
OUTPUT:
  jack:28
</code></pre>
<h4 id="end模块"><a class="markdownIt-Anchor" href="#end模块"></a> END模块</h4>
<p>当所有记录都处理完了后就会执行END模块的代码，一般用来输出一个结果，比如累加、数组等</p>
<pre class="highlight"><code class>CMD:
  cat data.txt |awk -F&quot;[, ]+&quot; 'BEGIN{sum=0} {sum+=$3} END{print sum/NR}'
OUTPUT:
  25.6
</code></pre>
<h3 id="awk数组"><a class="markdownIt-Anchor" href="#awk数组"></a> awk数组</h3>
<pre class="highlight"><code class="shell">CMD:
  echo |awk 'BEGIN{arr["a"]="aa";arr["b"]="bb";arr["c"]="cc"}END{for (key in arr) {print(arr[key])}}'
OUTPUT:
  aa
  bb
  cc
</code></pre>
<p>awk中的任何变量都不需要申明，直接使用即可</p>
<h2 id="实例"><a class="markdownIt-Anchor" href="#实例"></a> 实例</h2>
<h3 id="一个自动登录程序"><a class="markdownIt-Anchor" href="#一个自动登录程序"></a> 一个自动登录程序</h3>
<p>数据如下：</p>
<blockquote>
<p>1   machine1    username1    host1    password1<br>
2   machine2    username2    host2    password2<br>
3   machine3    username3    host3    password3</p>
</blockquote>
<pre class="highlight"><code class="sh"><span class="hljs-meta">#!/bin/sh</span>
PASSFILE=<span class="hljs-string">"./passwd.data"</span>
key=<span class="hljs-variable">$1</span>
cat <span class="hljs-variable">$PASSFILE</span>| awk -F<span class="hljs-string">"[\t ]+"</span> <span class="hljs-string">'$0 ~ /'</span><span class="hljs-string">"<span class="hljs-variable">$key</span>"</span><span class="hljs-string">'/ {print $1,$2,$3,$4}'</span>
<span class="hljs-built_in">printf</span> <span class="hljs-string">"please choose one: "</span>
<span class="hljs-built_in">read</span> choose
<span class="hljs-keyword">if</span> [[ <span class="hljs-string">"<span class="hljs-variable">$choose</span>"</span> != <span class="hljs-string">""</span> &amp;&amp; <span class="hljs-variable">$choose</span> -ge 0 ]]
<span class="hljs-keyword">then</span>
    <span class="hljs-built_in">eval</span> $(cat <span class="hljs-variable">$PASSFILE</span>| awk -F<span class="hljs-string">"[\t ]+"</span> <span class="hljs-string">'$1 == '</span><span class="hljs-string">"<span class="hljs-variable">$choose</span>"</span><span class="hljs-string">' {printf("arr=([1]=%s [2]=%s [3]=%s)",$3,$4,$5)}'</span>)

    user=<span class="hljs-variable">${arr[1]}</span>
    host=<span class="hljs-variable">${arr[2]}</span>
    pass=<span class="hljs-variable">${arr[3]}</span>

    <span class="hljs-comment"># 结合expect实现自动登录</span>
    <span class="hljs-built_in">echo</span> <span class="hljs-string">"ssh <span class="hljs-variable">${user}</span>@<span class="hljs-variable">${host}</span> &lt; <span class="hljs-variable">${pass}</span>"</span>
<span class="hljs-keyword">fi</span>
</code></pre>
<h3 id="统计文件单词出现次数"><a class="markdownIt-Anchor" href="#统计文件单词出现次数"></a> 统计文件单词出现次数</h3>
<pre class="highlight"><code class="sh">sed -r <span class="hljs-string">'s#[^a-zA-Z]+# #g'</span> data.txt |awk <span class="hljs-string">'BEGIN{RS="[ \t\n]+"} {print $0}'</span> |sort |uniq -c |sort -n
</code></pre>
<h3 id="统计与本机建立连接的ip机器连接数"><a class="markdownIt-Anchor" href="#统计与本机建立连接的ip机器连接数"></a> 统计与本机建立连接的ip机器连接数</h3>
<pre class="highlight"><code class="sh">netstat -an |grep ESTABLISHED |awk <span class="hljs-string">'{print $5}'</span> |awk -F: <span class="hljs-string">'{print $1}'</span> |sort |uniq -c |sort -nr
</code></pre>

</div>


    <div class="post-guide">
        <div class="item left">
            
              <a href="/2019/08/18/102.html">
                  <i class="fa fa-angle-left" aria-hidden="true"></i>
                  grep
              </a>
            
        </div>
        <div class="item right">
            
        </div>
    </div>




<script>
	
	
</script>
	</div>
	<div id="footer">
	<p>
	©2019-<span id="footerYear"></span> 
	<a href="/">wangxiaochuang</a> 
	
	
		|
		<span id="busuanzi_container_site_pv">
			pv
			<span id="busuanzi_value_site_pv"></span>
		</span>
		|
		<span id="busuanzi_container_site_uv"> 
			uv
			<span id="busuanzi_value_site_uv"></span>
		</span>
	
	</p>
</div>
<script type="text/javascript"> 
	document.getElementById('footerYear').innerHTML = new Date().getFullYear() + '';
</script>

	<button id="totop-toggle" class="toggle"><i class="fa fa-angle-double-up" aria-hidden="true"></i></button><!-- hexo-inject:begin --><!-- hexo-inject:end -->
</body>
</html>